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Fleming  and  his  associates  conducted  research  on  the  following 
topics: 

a)  Stochastic  control  under  partial  observations.  For  the  case  of 
partially  observed  diffusion  processes  a  study  of  "relaxed"  controls  and 
of  an  associated  separated  control  problem  (certainty-equivalence  prin¬ 
ciple)  was  made  in  Ref.  [1].  A  separated  control  problem  for  a  general 
class  of  controlled,  partially  observed  Markov  processes  appears  in  [2]. 
Connections  with  adaptive  control  of  Markov  processes  are  under  contin¬ 
uing  study. 

b)  Nonlinear  filtering.  Analytical  and  numerical  techniques  for 
approximately  optimal  nonlinear  filters  are  under  study  by  Fleming's 
Ph.D.  student  R.  McGwier.  The  analytical  results  take  the  form  of  a 
regular  perturbation  expansion,  while  the  numerical  results  are  based 
on  moving  finite  elements.  Preliminary  results  are  given  in  [3]  with 
details  to  appear  in  McGwier 's  thesis. 

c)  Large  deviations  for  nearly  deterministic  processes.  Results 
of  Ventsel-Freidl in  type  for  nearly  deterministic  processes  of  diffusion 
or  jump  type  are  obtained  in  Sheu's  thesis  [2],  using  stochastic  control 
methods.  In  improved  form  they  appear  in  [4,5]. 

d)  Transition  and  invariant  densities  for  nearly  deterministic 
diffusions.  Asymptotic  expansions  for  transition  densities  were  obtained 
in  [6].  These  are  a  step  toward  the  more  difficult  problem  of  justifying 
an  expansion  for  invariant  densities,  in  terms  of  a  small  parameter 
describing  the  variance  of  noise  driving  the  diffusion  process.  Partial 
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results  on  the  latter  problem  have  been  obtained  by  Sheu. 

Fleming  presented  a  plenary  address  [7]  at  the  1983  International 
Congress  of  Mathematicians. 


References 

1.  W.H.  Fleming  and  M.  Nisio,  On  stochastic  relaxed  controls  for 
partially  observed  diffusions,  Osaka  Math.  J.  (to  appear). 

2.  S-J  Sheu,  Optimal  control  and  its  application  to  large  deviation 
theory,  Ph.D.  Thesis,  Brown  University  1982. 

3.  W.H.  Fleming  and  R.  McGwier,  A  regular  perturbation  expansion  in 
nonlinear  filtering,  Proc.  22nd  IEEE  Co nf.  on  Decision  and  Control, 
Dec.  1983. 

4.  S-J  Sheu,  Stochastic  control  and  exit  probabilities  of  jump  processes 
to  appear  in  SIAM  J.  on  Control  and  Optimization. 

5.  S-J  Sheu,  Stochastic  control  and  principal  eigenvalue.  Stochastics 
Vol .  11  (1984),  pp.  191-211. 

6.  S-J  Sheu,  Asymptotic  expansion  for  transition  density  of  diffusion 
Markov  process  with  small  diffusion,  to  appear  in  Stochastics. 

7.  W.H.  Fleming,  Optimal  control  of  Markov  processes,  Proc.  Internat. 
Congress  Math,  Warsaw,  1983. 


Kushner  completed  a  major  monograph  on  approximation  methods  in 
stochastic  systems  theory.  The  book  develops  several  very  useful  methods 
and  discusses  many  complicated  and  realistic  practical  applications. 

The  methods  are  powerful  and  useful  for  the  analysis  of  both  discrete 
parameter  and  continuous  parameter  systems  with  wide  band  width  random 
inputs  or  disturbances,  and  develops  a  theory  of  stability  for  'near' 
Markovian  systems.  The  major  idea  is  that  the  typical  system  is  hard 
to  analyze  -  so  one  must  seek  'computable'  and  'reasonable'  approxima¬ 
tions.  The  book  goes  about  this  in  a  systematic  and  thorough  fashion. 
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APPROXIMATION  AND  WEAK  CONVERGENCE 
METHODS  FOR  RANDOM  PROCESSES,  WITH 
APPLICATIONS  TO  STOCHASTIC  SYSTEMS  THEORY 
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Division  of  Applied  Mathematics  and  Engineering!131^ 

Brown  University  . 

Providence,  Rhode  Island  02912  I 

I .  Introduction. 

This  book  is  concerned  with  methods  for  approximating  'physical' 
random  processes  by  the  much  more  tractable  diffusion  or  jump-diffusion 
processes,  and  with  their  applications  to  modelling,  approximation  and 
analysis  problems  in  stochastic  control  and  communication  theory. 
Although  we  specialize  the  applications  to  these  areas,  the  methods 
are  of  general  applicability  to  similar  modelling  and  approximation 
problems  in  many  areas  of  physics,  engineering  and  operations  research. 
Many  of  the  results  are  new,  and  other  known  results  have  been  modi¬ 
fied  to  enhance  their  usefulness  for  the  types  of  applications  of 
interest. 

In  order  to  motivate  one  particular  type  of  problem  which  is  of 
interest,  consider  the  frequently  encountered  canonical  model 
2  =  b(z)  +  o(z)E,  where  £{•)  is  a  random  process.  It  is  a  'physical' 

I 

process,  in  the  sense  that  it  is  obtained  more  or  less  directly  from 
the  physical  laws  underlying  the  application.  Suppose  that  (loosely 
speaking)  it  fluctuates  'much  more  rapidly'  than  does  z(-)»or  has  a 
wide  band-width.  In  order  to  facilitate  analysis,  and  owing  to  the 
large  assumed  differences  in  the  'rate  of  fluctuation'  of  £;(•)  and 
z(*).  it  is  common  practice  in  applications  to  replace  £(•)  by  (say) 
a  Gaussian  white  noise  (with  or  without  a  'correction*  term),  or  to 
find  a  suitable  white  noise  model  which  approximates  z(-)  in  some  sense. 
With  such  a  replacement,  z(*)  is  (loosely  speaking)  turned  into  a 
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Markov  process,  a  solution  to  a  stochastic  differential  equation,  and  a 
large  number  of  powerful  analytical  and  numerical  tools  can  be  applied 
to  it.  It  is  the  availability  of  such  tools  for  use  on  the  diffusion 
(or  jump-diffusion)  process  that  makes  the  approximations  so  important 
and  so  appealing  in  practice,  since  the  process  z(-)  will  usually  be 
very  hard  to  study.  Such  approximation  ideas  are  used  in  numerous  areas 
of  science  and  engineering,  and  a  great  deal  has  been  learned  in  recent 
years  on  the  appropriate  methodology  and  on  the  dependence  of  the  func¬ 
tions  in  the  approximation  on  the  dynamics  of  the  'physical'  process. 

One  looks  for  nice  approximating  processes  x(*)  such  that  the  distri¬ 
butions  of  important  quantities  are  close  to  those  of  z(-)- 

Sometimes,  in  applications,  the  approximations  are  just  written 
down  following  a  rough  heuristic  argument.  This  might  work  well  (and 
even  be  validated  by  mathematical  analysis)  for  simple  problems,  where 
the  dynamical  terms  are  smooth  and  £(•)  appears  in  a  simple  way.  But, 
if  the  structure  of  ?{•)  is  complicated,  if  it  depends  on  z(-)»  or 
if  a(x)  depends  on  x  or  b(x)  is  discontinuous,  or  if  the  equation 
is  of  the  more  complex  type  z  =  b(z,£),  where  b(-,-)  might  have  a 
complicated  structure,  then  simple  heuristic  procedures  can  often  yield 
incorrect  results,  or  even  be  unavailable.  In  general,  they  do  not 
provide  a  methodology  which  one  can  use  with  confidence  in  difficult 
problems.  Even  in  typical  cases,  the  systems  of  interest  can  be  quite 
complicated,  from  the  modelling,  analysis  or  approximation  point  of 
view.  They  can  contain  discontinuous  functions  of  the  state  and  noise 
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(for  example  the  limiter  function  sign  (•))•  the  noise  need  not  be 
stationary,  it  can  depend  on  z(-)  (be  a  product  of  the  feedback  in 
the  system)  and  it  might  occur  in  the  dynamical  equations  in  a  compli¬ 
cated  non-linear  fashion  with  the  state.  Similarly  for  the  discrete  param¬ 
eter  cases.  In  fact,  the  discrete  parameter  cases  can  be  the  more  com¬ 
plex,  since  they  often  arise  from  the  use  of  recursive  algorithms  for 
estimation,  etc.,  and  can  involve  rather  non-linear  digital  processing 
of  the  data. 

Any  theory  of  approximation  involves  the  convergence  of  a  paramet¬ 
rized  sequence  (xe( - ) >  of  processes  to  a  limit  process  in  some  specific 
sense.  To  put  the  above  mentioned  motivational  problem  in  this  form, 
parametrize  the  bandwidth  or  'rate  of  fluctuations'  of  £(•)  by  e;  as 
e  goes  to  zero  the  bandwidth  goes  to  infinity.  Write  z(-)  and  £(-) 
as  xe(- )  and  £e(-)t  resp.  Roughly,  we  imbed  the  true  physical  process 
in  a  suitable  parametrized  family,  and  then  find  a  process  x(-)  such 
that  xe(-)  -*■  x(-)  in  some  specific  sense.  Hopefully,  the  limit  process 
will  be  (  a  diffusion  or  jump-diffusion,  degenerate  or  not) 
much  more  tractable  mathematically  or  numerically  than  is  the  true  phy¬ 
sical  process  z(*).  and  that  the  parameter  value  e  corresponding  to 
this  physical  process  will  actually  be  small  enough  so  that  the  approxi¬ 
mation  is  good.  (The  bandwidth  concept  was  used  for  motivation  only;  it 
need  not  be  defined  or  meaningful  for  the  processes  in  the  book.) 

The  most  widely  used  and  useful  sense  of  limit  is  that  of  weak  con¬ 
vergence  of  measures  (Chapter  2).  This  is  a  powerful  generalization  of 


the  notion  of  convergence  of  multivariate  distributions,  and  it  provides 
the  data  and  approximations  of  interest.  Such  approximations  are  of 
interest  for  both  the  analysis  and  synthesis  of  systems.  The  weak  con¬ 
vergence  based  limit  theorems  have  certain  features  in  common  with  the 
usual  central  limit  theorem.  As  e  •>  0,  much  of  the  specific  detail  of 
the  system  is  often  wiped  out,  as  for  the  central  limit  theorem,  where 
as  the  number  of  summands  increases,  the  distribution  of  the  sum  depends 
'less  and  less*  on  the  character  of  each  summand,  and  is  'closer  and 
'closer'  to  the  normal  or  normal -Poisson  distribution.  Thus,  for  analy¬ 
sis  purposes,  the  limit  processes  are  useful  because  they  usually  have 
a  simpler  mathematical  or  physical  structure  than  the  original  processes, 
even  though  distributions  of  important  quantities  are  close.  For  pur¬ 
poses  of  synthesis,  approximations  provide  a  convenient  vehicle  for 
studying  the  effects  of  system  modifications.  The  weak  convergence 
techniques  are  widely  used  in  operations  research  and  in  statistics,  and 
are  now  of  increasing  interest  in  control  and  communication  theory,  owing 
to  their  usefulness  in  the  modelling,  simplification  and  study  of  complex 
non-linear  systems. 

Chapter  1  contains  a  number  of  useful  background  results  from  prob¬ 
ability  theory.  Many  of  the  more  difficult  parts  of  the  chapter  are  not 
actually  used  in  the  rest  of  the  book,  but  are  introduced  in  order  to 

allow  a  discussion  of  the  relationship  between  two  points  of  view  toward 

<>\ 

diffusion  processes.  The  first  is  as  the  solution  to  the  standard  Ito 
or  stochastic  differential  equation,  the  second  is  as  the  solution  to 
the  so-called  martingale  problem  of  Stroock  and  Varadhan  [if].  The 


second  point  of  view  is  perhaps  less  intuitive  for  physicists  and 
engineers,  but  has  many  advantages  in  weak  convergence  analysis;  in 
particular,  it  is  much  easier  to  show  that  a  process  solves  the  martin¬ 
gale  problem  than  it  is  to  show  directly  that  it  satisfies  a  stochastic 
differential  equation.  Because  of  this,  it  is  useful  to  understand  the 
equivalence  (under  broad  conditions)  of  the  two  points  of  view,  and  a 
lare  part  of  Chapter  1  is  devoted  to  a  heuristic  discussion  of  this 
equivalence,  and  to  the  background  material  on  martingales.  Only  a  few 
facts  from  this  chapter  will  be  used  in  the  sequel.  The  reader  who  is 
interested  mainly  in  the  methodology  in  applications  can  quickly  skim 
this  chapter.  A  number,  of  basic  definitions  and  useful  results  from 
the  theory  of  weak  convergence  are  given  in  Chapter  2. 

Three  general  methods  are  used  to  get  the  weak  convergence  and  iden¬ 
tify  the  limits,  the  perturbed  test  function  method,  the  direct  averaging 
method  and  a  method  which  combines  some  of  the  best  features  of  both, 
and  is  perhaps  the  most  generally  useful.  The  second  method  is  perhaps 
the  easiest  to  use,  in  its  domain  of  applicability.  The  first  is  dealt 
with  in  Chapters  3  and  4,  and  the  last  two  in  Chapter  5.  There  are 
roughly  two  main  steps  in  using  weak  convergence  theory;  checking  whether 
there  is  a  limit,  and  identifying  it  (and  verifying  its  uniqueness).  The 
compactness  or  tightness  criteria  which  we  use  to  check  whether  there  is 
a  limit  are  discussed  in  Chapter  3.  Then,  given  that  there  is  a  limit, 
we  must  characterize  it.  To  each  limit  process,  there  will  be  associated 
an  'infinitesimal  operator'  or  'differential  generator'  A.  If  the  limit 
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is  the  scalar  valued  diffusion  defined  by  the  1  to  equation 
dx  =  b(x)dt  +  o(x)dw,  then  A  =  b(x)3/3x  +  ^  a^(x) 3^/3x*\  If  to  each 
A,  there  is  a  unique  process,  then  knowing  A  is  equivalent  to  knowing 
the  process.  The  three  methods  use  different  ways  of  averaging  the 
noise  effects  to  obtain  A  from  the  sequence  xe(-)  and  its  'driving 
noises'  .  Once  the  main  steps  in  the  general  techniques  are  understood. 
They  are  of  value  even  for  formal  use,  since  they  provide  for  an  essen¬ 
tially  automatic  way  of  constructing  the  operator  A  of  the  limit 
process. 

We  try  to  present  the  ideas  and  results  in  a  general  enough  way  so 
that  they  can  be  used  in  many  interesting  applications,  but  without  even 
beginning  to  exhaust  the  possibilities. 

For  many  applications,  the  systems  are  in  operation  for  a  very  long 
time.  Weak  convergence  theory  in  itself  does  not  tell  us  much  about  what 
happens  to  xe(t)  for  small  e  and  large  t,  yet  such  information  is 
vital  to  many  applications.  The  weak  convergence  results  can  be  extended 
to  get  the  desired  information,  under  suitable  stability  properties  on 
(xG( - )>  (uniformly  in  e).  Chapter  6  obtains  usable  criteria  for  these 
stability  properties.  The  properties  of  xe(t)  for  large  t  and  small 
e  are  related  to  the  asymptotic  properties  of  x(t),  the  limit  diffusion. 
Invariant  measures,  'almost'  invariant  measures,  recurrence  and  moment 
boundedness  (uniformly  in  e),  and  similar  stability  properties  are  dis¬ 
cussed.  The  results  are  useful  more  generally  for  the  study  of  stability 
properties  of  non-Markovian  systems. 
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Some  problems  in  singular  perturbations  are  discussed  in  Chapter  7. 
Chapters  8  to  10  deal  with  numerous  applications;  adaptive  filters  and 
antenna  arrays,  stochastic  approximations,  adaptive  quantifiers,  input- 
output  statistics  of  important  non-linear  devices,  phase  locked  loops 
(with  and  without  limiters),  etc.  The  idea  is  to  show  how  to  use  the 
approximation  techniques  to  obtain  useful  diffusion  approximations  to  a 
aanonical  problem  in  each  class,  so  as  to  facilitate  its  analysis. 

Those  applications  will  attest  to  the  power  and  usefulness  of  the  tech¬ 
niques  for  obtaining  information  on  hard  problems,  often  with  only  a 
reasonable  amount  of  work. 

Chapter  11  deals  with  an  approximation  problem  which  is  somewhat 
different  from  those  dealt  with  in  the  previous  chapters.  It  concerns 
systems  which  are  of  interest  for  a  long  period  of  time,  but  where  the 
noise  effects  are  small.  A  typical  'classical'  problem  in  this  area 
concerns  the  'small  white  noise'  system  dxE  =  F(xe)dt  +  /Eadw,  where 
e  is  small,  and  w(-)  is  a  Wiener  process.  Let  x  =  F(x)  be  asymp¬ 
totically  stable  at  x  =  0.  In  many  applications  in  physics  and  engin¬ 
eering,  one  is  interested  in  mean  escape  times  from  a  neighborhood  of 
0,  or  in  similar  'escape  time'  statistics.  Since  this  is,  in  general, 
a  ver  hard  problem,  asymptotic  methods  are  useful,  and  one  seeks  the 
statistics  (normalized  in  a  suitable  way)  as  e  -*■  0.  A  non-ideal  or 
practical  system  would  be  xE  =  F(xG)  +  c(xE)£E(t),  where  £c(*)  is 
scaled  so  that  its'  effects  are  small.  The  computation  is  substantially 
easier,  however,  with  the  'white  noise'  ideal  system.  The  general  ideas 


behind  the  estimates,  for  both  the  non-ideal  and  the  ideal  system  are 
treated.  The  proper  theoretical  foundation  is  in  the  theory  of  large 
deviations.  An  introduction  to  the  relevant  part  of  the  theory  is  pro 
vided,  and  several  examples  are  discussed  to  illustrate  when  one  can 
or  cannot  use  the  small  white  noise  ideal  models. 


